Phrase based document clustering with automatic phrase extraction

ABSTRACT

Meaningful phrases are distinguished from chance word sequences statistically, by analyzing a large number of documents and using a statistical metric such as a mutual information metric to distinguish meaningful phrases from groups of words that co-occur by chance. In some embodiments, multiple lists of candidate phrases are maintained to optimize the storage requirement of the phrase-identification algorithm. After phrase identification, a combination of words and meaningful phrases can be used to construct clusters of documents.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.12/785,105, filed May 21, 2010, which claims the benefit of U.S.Provisional Application No. 61/300,385, filed Feb. 1, 2010, which areboth hereby incorporated by reference.

BACKGROUND

The present invention relates in general to semantic clustering ofdocuments and in particular to semantic clustering using a combinationof words and multi-word phrases that may appear in the document.

With the proliferation of computing devices and communication networkssuch as the Internet, an ever increasing amount of information is storedin the form of electronic documents. Such documents might be generatedusing application software such as word processing programs, e-mailprograms, web page development tools, etc. Electronic documents can alsobe generated by scanning paper documents and employing optical characterrecognition (“OCR”) or other techniques to create an electronicrepresentation of the content.

It is often necessary to search through a large collection of electronicdocuments to find information relevant to a particular question. Forexample, a number of search services provide interfaces via which userscan search electronic documents that are accessible via the World WideWeb. In another context, discovery in civil litigation usually involvesthe production of massive quantities of electronic documents that theproducing and receiving parties must sift through.

To facilitate review of a large corpus of documents, a number ofanalysis techniques have been developed that automatically determineproperties of the document, e.g., by analyzing the patterns ofoccurrence of words. For example, semantic clustering attempts to groupdocuments pertaining to the same topic, generally based on identifyingwords or combinations of words that tend to occur in documents withinthe cluster but not in documents outside the cluster.

One difficulty in semantic clustering is that many languages (such asEnglish) include multi-word groups (phrases) that convey a meaning to auser. The meaning of such phrases can be different from the singlewords. For example “New York” and “ice cream” are recognized phrases.Human readers recognize such phrases, but computers do not. Semanticclustering algorithms based on single words can thus be missingimportant pieces of information, leading to less accurate results.

To address this, some efforts have been made to incorporate phraseidentification into semantic clustering. For example, some clusteringprograms provide a list of phrases, and sequences of words fromdocuments can be compared to the list to detect phrases. This form ofphrase detection is limited to those phrases that happen to be on thelist. Other clustering programs use punctuation cues (e.g., capitalletters) to identify phrases; this works well for proper nouns such as“New York” or “Frank Sinatra” but not for phrases such as “ice cream”that are not normally capitalized.

It would therefore be desirable to automate the process of identifyingmeaningful phrases within documents or collections of documents.

SUMMARY

In certain embodiments of the present invention, meaningful phrases aredistinguished from chance word sequences statistically, by analyzing alarge number of documents and distinguishing word sequences that occurmore often than random sampling would predict from other word sequences.In some embodiments, a mutual information metric (or comparablestatistical metric) is defined and computed for multi-word sequencesthat are possible phrases, and the meaningful phrases are distinguishedfrom chance sequences based on the mutual information metric. Acombination of words and meaningful phrases (identified statistically)can be used to construct clusters of documents.

Keeping track of candidate phrases is a challenging problem. Forinstance, a large corpus (e.g., a million documents) may include tens orhundreds of thousands of distinct words, and possible two-word phrasescan number in the tens of millions. For three-word phrases, the numbersare even more staggering. Some embodiments of the present inventionreduce the data management burden by maintaining two (or more) candidatephrase lists. In an embodiment with two lists, one list is used to keeptrack of candidate phrases that have occurred once, the other to keeptrack of candidate phrases that have occurred multiple times. When acandidate phrase is encountered for the first time, it is added to thefirst list; if it occurs again, it is moved to the second list and anoccurrence count is maintained. To keep storage requirements withinmanageable bounds, the first list can be limited to a maximum number ofcandidate phrases (e.g., on the order of one million). Once this limitis reached, each time a new candidate phrase is added to the first list,an older candidate phrase is dropped from the list; the candidate phraseto be dropped can be selected randomly, pseudorandomly or by otherselection algorithms such as least recently added.

One aspect of the present invention relates to methods of extractingphrases from a corpus of documents. A processor can generate a set ofcandidate phrases from the documents in the corpus, where each candidatephrase corresponds to a group of two or more words that occurconsecutively in at least one of the documents in the corpus and eachcandidate phrase has an associated occurrence count. The processor cancompute a statistical metric for each candidate phrase based at least inpart on the occurrence count; the statistical metric can be any metricthat indicates a likelihood of the words within the candidate phraseoccurring consecutively by chance. Based on the statistical metric, theprocessor can select some (or potentially all) of the candidate phrasesas meaningful phrases.

In some embodiments, the method includes keeping two lists of candidatephrases. When a group of consecutive words from one of the documents inthe corpus, a determination is made as to whether the group ofconsecutive words appears as one of the candidate phrases in a firstlist of candidate phrases. If so, then an occurrence count associatedwith that candidate phrases is incremented. If not, a determination ismade as to whether the group of consecutive words appears as one of thecandidate phrases in a second list of candidate phrases. If so, thenthat candidate phrase is promoted to the first list (with an occurrencecount of 2); if not, then the group of consecutive words is added as anew candidate phrase to the second list. An upper limit (e.g., onemillion phrases) can be imposed on the number of candidate phrases inthe second list. If the limit is reached, then a candidate phrase fromthe list can be deleted each time a new phrase is to be added. Thephrase to be deleted can be selected randomly, pseudorandomly, oraccording to some other selection algorithm.

In some embodiments, the method can also include forming clusters ofdocuments. For example, document vectors can be constructed fordocuments from the corpus. The document vector for each document caninclude some components that correspond to words and other componentsthat correspond to some or all of the meaningful phrases. Documents canbe clustered based on similarity of the vectors (e.g., using knowntechniques for comparing document vectors).

In some embodiments, the method can also include assigning names to theclusters. For example, the most frequently occurring terms in thecluster (where “term” refers to either a word or a meaningful phrase)can be identified as candidate terms for the name. If a word appears inmore than one of the candidate terms, then a single term containing theword can be selected as a candidate term. After de-duplicating thecandidate terms (so that no word appears in more than one term), some ofthe candidate terms can be selected for inclusion in the name for thecluster. This selection can be based on weights associated with thecandidate terms. In some embodiments, the resulting cluster nameincludes multiple terms, with no word appears in more than one of theterms.

Another aspect of the invention relates to computer readable storagemedia containing program instructions that, when executed by a computersystem cause the computer system to execute a method of forming documentclusters from a corpus of documents. For example the computer system cangenerate a set of candidate phrases from the documents in the corpus,with each candidate phrase corresponding to a group of two or more wordsthat occur consecutively in at least one of the documents in the corpus.The computer system can compute a mutual information metric for eachcandidate phrase based on one or more occurrences of the candidatephrase and one or more separate occurrences of the words within thecandidate phrase. Based on the mutual information metric, meaningfulphrases can be selected from the set of candidate phrases, the selectionbeing based on the mutual information metric. Occurrence patterns of themeaningful phrases and single words in the documents can be used to formdocument clusters.

Another aspect of the invention relates to a computer system. Thecomputer system can include a storage subsystem and a processor coupledto the storage subsystem. The processor can be configured, e.g., byproviding suitable program code, to extract and store in the storagesubsystem a set of candidate phrases from the corpus of documents, whereeach candidate phrase corresponds to a group of two or more words thatoccur consecutively in at least one of the documents in the corpus. Theprocessor can also be configured to compute a statistical metric foreach of the candidate phrases based on occurrence count data indicatinga number of occurrences of the candidate phrase in documents of thecorpus and a number of occurrences of the words making up the candidatephrase; any statistical metric can be used that indicates a likelihoodof the words within candidate phrase occurring consecutively by chance.The processor can also be configured to select one or more phrases fromthe set of candidate phrases as a meaningful phrase, based on thestatistical metric and to store a list of the meaningful phrases in thestorage subsystem.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system according to anembodiment of the present invention.

FIG. 2 is a flow diagram of a process for phrase-based clusteringaccording to an embodiment of the present invention.

FIG. 3 is a flow diagram of a phrase extraction process according to anembodiment of the present invention.

FIG. 4 illustrates n-tuples for a representative text according to anembodiment of the present invention.

FIG. 5 illustrates an optimized phrase list structure according to anembodiment of the invention.

FIG. 6 is a flow diagram of a process for managing the phrase liststructure of FIG. 5 according to an embodiment of the invention.

FIG. 7 is a flow diagram of a process for automatically generating acluster name according to an embodiment of the present invention.

DETAILED DESCRIPTION

In certain embodiments of the present invention, meaningful phrases aredistinguished from chance word sequences statistically, by analyzing alarge number of documents and distinguishing word sequences that occurmore often than random sampling would predict from other word sequences.In some embodiments, a mutual information metric (or comparablestatistical metric) is defined and computed for multi-word sequencesthat are possible phrases, and the meaningful phrases are distinguishedfrom chance sequences based on the mutual information metric. Acombination of words and meaningful phrases (identified statistically)can be used to construct clusters of documents.

Keeping track of candidate phrases is a challenging problem. Forinstance, a large corpus (e.g., a million documents) may include tens orhundreds of thousands of distinct words, and possible two-word phrasescan number in the tens of millions. For three-word phrases, the numbersare even more staggering. Some embodiments of the present inventionreduce the data management burden by maintaining two (or more) candidatephrase lists. In an embodiment with two lists, one list is used to keeptrack of candidate phrases that have occurred once, the other to keeptrack of candidate phrases that have occurred multiple times. When acandidate phrase is encountered for the first time, it is added to thefirst list; if it occurs again, it is moved to the second list and anoccurrence count is maintained. To keep storage requirements withinmanageable bounds, the first list can be limited to a maximum number ofcandidate phrases (e.g., on the order of one million). Once this limitis reached, each time a new candidate phrase is added to the first list,an older candidate phrase is dropped from the list; the candidate phraseto be dropped can be selected randomly, pseudorandomly, or by some otherselection algorithm such as least recently added.

System Overview

FIG. 1 is a block diagram of a computer system 100 according to anembodiment of the present invention. Computer system 100 includes a CPU102, storage subsystem 104, network interface 106, and user interface108 connected via a bus 110. CPU 102 can be, e.g., any programmablegeneral-purpose processor. Network interface 106 provides access to oneor more other computer systems via a network 112, which can include,e.g., a local area network (LAN), a wide area network (WAN), theInternet (a globally interconnected network of computer networks), avirtual private network, and so on. Network interface 106 can beimplemented using standard protocols, including wired protocols (e.g.,Ethernet) and/or wireless protocols (e.g., any IEEE 802.11 protocol).User interface 108 can include one or more input devices 114 such as akeyboard, mouse, touch screen, touch pad, etc., and one or more outputdevices such as a display 116. Bus 110 can be implemented usingconventional bus architectures and may include bridges, bus controllers,and the like.

Storage subsystem 104 incorporates various computer-readable storagemedia to provide storage for programs and data accessed by CPU 102and/or other components of computer system 100. In the embodiment shown,storage subsystem 104 includes primary memory 118. Primary memory 118provides the fastest access times and can be implemented using knownmemory technologies such as DRAM (dynamic random access memory) and/orSRAM (static random access memory). Primary memory 118 is advantageouslyused at any given time to store programs and/or data that are activelyin use by CPU 102. Thus, for example, memory 118 is shown as storing aclustering program 120 that, when executed, causes CPU 102 to generateclusters from documents in the corpus. Memory 118 in this example alsostores a phrase extraction program 121 that, when executed, causes CPU102 to identify meaningful phrases based on statistical usage patternsin the document corpus. These phrases can be used within clusteringprogram 120. Examples of phrase extraction and phrase-based clusteringare described below.

Storage subsystem 104 in this embodiment also provides various secondarystorage areas, which can include, e.g., magnetic media such asconventional hard or floppy disks, optical media such as compact disc(CD), digital versatile disc (DVD), or the like, and/or semiconductormedia such as flash memory. Secondary storage areas generally havelonger access time than primary memory 118 but have larger storagecapacity. In this example, secondary storage areas are provided for ananalysis data store 130 and a document information data store 124.

Document information data store 124 provides information (also referredto as metadata) about a corpus of documents. As used herein, a “corpus”of documents can be any collection of documents about which informationis to be provided to a user of system 100. In one embodiment, the corpusof documents (or a portion thereof) can be stored in a documentrepository 126 that is remote from computer system 100 and accessiblevia network interface 106 and network 112. In another embodiment, thecorpus (or a portion thereof) can be stored locally, e.g., withinstorage subsystem 104. The corpus can be centralized or distributed(e.g., it can be a collection of World Wide Web documents that arestored on respective web servers connected to network 112 as is known inthe art) as desired, and document information data store 124 might ormight not contain actual documents.

Document information data store 124 can include a document record 125for each document in the corpus. Document record 125 can include, e.g.,a unique identifier of the document (“DocID”) and metadata about thedocument, including for example identifiers for any clusters to whichthe document has been assigned by clustering program 120. Otherinformation about the documents can also be stored, such as date ofcreation, editing, and/or addition to the corpus; type of document(e.g., e-mail, web page, word processor document); author; source orlocation from which the document was obtained; a condensedrepresentation of document content in a readily searchable form;language information; keywords; and so on.

Document information data store 124 can also include a cluster map 127that provides lists of document identifiers associated with each clustergenerated by clustering program 120. Other information about theclusters, such as a cluster name, a relationship (e.g., hierarchicalrelationship) to other clusters, and so on, can also be included incluster map 127.

In some embodiments, highly similar documents may be treated as the samedocument for purposes of clustering, and document information data store124 can be organized such that highly similar documents are groupedtogether. An example of such an implementation is described incommonly-assigned co-pending U.S. Patent Application No. 61/300,382filed of even date herewith.

Analysis data store 130 in this embodiment provides data that can bereferenced by programs executing on CPU 102, e.g., phrase extractionprogram 121 and/or clustering program 120. For example, analysis datastore 130 can include a dictionary 132. As used herein, a “dictionary”can include any list of words (i.e., character strings) in any languageor combination of languages, and the list can include any number ofwords. Dictionary 132 can be used to define a “term space” for purposesof characterizing a document. Analysis data store 130 can also provide aphrase list 134, which can include meaningful phrases that can be usedin clustering. In some embodiments, phrase list 134 is populatedautomatically through execution of phrase extraction program 121. Atvarious times, analysis data store 130 can also include other data suchas cluster definitions or the like that may be useful in analyzingdocuments.

It will be appreciated that computer system 100 is illustrative and thatvariations and modifications are possible. For example, although storagesubsystem 104 is shown as local to system 100, in alternativeembodiments, all or part of storage subsystem 104 can be implemented asremote storage, e.g., on a storage area network (SAN) or other remoteserver accessible via network 112. Thus, for example, documentinformation data store 124 and/or analysis data store 130 can be storedlocally or remotely as desired. Further, although clustering program 120and phrase extraction program 121 are shown as residing in primarymemory 118, the invention is not limited to any particular mechanism forsupplying program instructions for execution by CPU 102. For instance,at any given time some or all of the program instructions for clusteringprogram 120 or phrase extraction program 121 may be present within CPU120 (e.g., in an on-chip instruction cache and/or various buffers andregisters), in a page file or memory mapped file on a system disk,and/or in other local or remote storage space. In some embodiments,computer system 100 might be implemented as a server accessible to auser via a network, and user interface 108 is optional. Computer system100 may also include additional components such as floppy disk drives,optical media drives (e.g., CD or DVD), network interface components,USB interface, and the like. Computer system 100 can be configured withmany different hardware components and can be made in many dimensionsand styles (e.g., laptop, tablet, server, workstation, mainframe);network connections may be provided via any suitable transport media(e.g., wired, optical, and/or wireless media) and any suitablecommunication protocol (e.g., TCP/IP). A particular computerarchitecture or platform is not critical to the present invention.

Phrase-Based Clustering Overview

FIG. 2 is a flow diagram of a process 200 for phrase-based clusteringaccording to an embodiment of the present invention. In process 200, acorpus of documents can be processed twice: once to identify meaningfulphrases and again to form clusters using words and phrases.

Process 200 starts (block 202) with a corpus of documents that can beprovided in various ways (e.g., via document collection processes or Webcrawlers). At block 204, the corpus is processed to identify meaningfulphrases based on statistical properties of the documents in the corpus;examples of algorithms that can be used are described below. At block206, a document vector is constructed for each document using acombination of words and meaningful phrases (as identified at block204). At block 208, clusters are generated based on the documentvectors. Known clustering algorithms, such as those described in U.S.Pat. No. 7,469,246 and U.S. Pat. No. 7,308,451, or other algorithms canbe used.

At block 210, a name is generated for each cluster. The name canincorporate both words and phrases that frequently occur in the vectorsof documents in the cluster. An example of name generation using wordsand phrases as terms is described below. Thereafter, process 200 can end(block 212).

It will be appreciated that process 200 is illustrative and thatvariations and modifications are possible. The following sectionsdescribe specific implementations of various blocks within process 200.

Phrase Extraction

As noted above, meaningful phrases can be automatically extracted from acorpus of documents using statistical analysis. Ideally, a meaningfulphrase is one that conveys a distinct meaning to a person when itsconstituent words are read as a unit. For example, “New York” and “canof worms” are meaningful phrases to English speakers; in this case, themeaning is something other than the sum of the individual words. In somecases, a phrase can be meaningful by virtue of narrowing a broad fieldin a helpful way; for instance, “computer security” and “airportsecurity” identify specific (and very different) areas within the broadfield of “security.”

In embodiments of the present invention, phrases are extracted usingstatistical properties, and a “meaningful” phrase as used herein refersgenerally to any phrase that passes statistical tests (e.g., asdescribed below) indicating that it is likely to convey a distinctmeaning to a person, rather than being a chance conjunction of words. Itshould be noted that not all phrases identified as meaningful by thetechniques described herein need actually convey a distinct meaning to aperson.

FIG. 3 is a flow diagram of a phrase extraction process 300 according toan embodiment of the present invention. Process 300 starts (block 302)with a corpus of documents to be analyzed. At block 304 a first documentis selected. At block 306, a list of “n-tuples” of words occurring inthe document is generated. As used herein, an “n-tuple” is a set of nconsecutive words for some integer n. In various embodiments, block 306can include parsing the document into tokens (words) using conventionaltoken-extraction software, then generating a list of n-tuples from thetokens. A 1-tuple is a single word, a 2-tuple is a pair of consecutivewords, and a 3-tuple is a trio of consecutive words. Thus, for example,in “The quick brown fox jumps over the lazy dog,” “quick” would be a1-tuple, “quick brown” would be a 2-tuple, and “quick brown fox” wouldbe a 3-tuple. In some embodiments, all n-tuples are generated for nrunning from 1 to n_(max); in one such embodiment, n_(max)3. A list ofn-tuples for “The quick brown fox jumps over the lazy dog” is shown inFIG. 4 for n_(max)=3. Single words (1-tuples) are listed in box 402,two-word sequences (2-tuples) in box 404, and three-word sequences(3-tuples) in box 406.

In some embodiments, punctuation can be taken into consideration whengenerating n-tuples for n>1. For example, meaningful phrases typicallydo not include periods or commas (or certain other punctuation marks)between the words. Accordingly, n-tuple extraction can use punctuationmarks as barriers. Thus, for example the words “blue fox” would not beextracted as a 2-tuple if they appeared as “blue. Fox” but would be ifthey appeared as “blue fox” with no intervening punctuation.

At block 308, the list of n-tuples is cleaned up. For example, 1-tuplesthat are stopwords and any larger n-tuples that begin or end in astopword can be removed from the list. As is known in the art,“stopwords” are terms used with such high frequency that they aregenerally unhelpful in statistical analysis of document content.Examples of stopwords include indefinite articles (e.g., “a,” “an”),definite articles (e.g., “the,” “this,” “that”), and common prepositions(e.g., “of”). In one embodiment, a list of stopwords can be stored indictionary 132 (FIG. 1) or elsewhere, and stopwords in the n-tuples canbe identified by checking against the list.

By way of example, FIG. 4 illustrates the effect of removal of stopwordsin one embodiment. In this embodiment, stopword removal transforms list402 to list 412, list 404 to list 414, and list 406 to list 416. Notethat in this embodiment, 3-tuples with stopwords in the middle (e.g.,“over the lazy”) are not removed. This allows detection of phrases suchas “can of worms.”

At block 310, for the remaining n-tuples, the number of occurrences ofeach n-tuple is determined. At block 312, for n-tuples with n>1, aphrase list is updated. The phrase list keeps track of candidate phrasesthat have been found in documents and the number of occurrences of eachcandidate phrase.

One of the difficulties in extracting meaningful phrases lies in therapidity with which the number of candidate phrases increases asdocuments are analyzed. For example, a large document corpus (e.g., amillion documents) might contain hundreds of thousands of unique words,but tens of millions of unique phrases. Storing all unique n-tuples ascandidate phrases can be done, given enough disk space; however, someembodiments reduce the storage requirement through careful management ofphrase lists, e.g., as described below.

At block 314, it is determined whether more documents remain to beprocessed. If so, process 300 returns to block 304 to select the nextdocument.

After all documents have been processed, at block 316 phrase statisticsfor the corpus are computed. In some embodiments, the phrase statisticsinclude a mutual information metric that indicates likelihood that thewords in a particular candidate phrase appeared together by chance. Forexample, the mutual information between two random variables X (withpossible values x_(i)) and Y (with possible values y_(j)) can be definedas:

$\begin{matrix}{{{I\left( {X:Y} \right)} = {\sum\limits^{\;}{{p\left( {x_{i},y_{j\;}} \right)}\log\frac{p\left( {x_{i},y_{j}} \right)}{{p\left( x_{i} \right)}{p\left( y_{j} \right)}}}}},} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$where p(x_(i)) is the probability of value x_(i), p(y_(j)) is theprobability of y_(j), p(x_(i), y_(j)) is the probability of x_(i) andy_(j) co-occurring, and the sum is taken over all combinations of(x_(i), y_(j)).

In the case of a candidate phrase with two words A and B, Eq. (1) can beused, with the possible values of X being A or non-A (denoted ˜A herein)and possible values of Y being B or non-B (denoted ˜B herein).Accordingly, p(A) is the number of occurrences of word A divided by thetotal number of (non-unique) words in the corpus, p(B) is the number ofoccurrences of word B divided by the total number of words in thecorpus, and p(A, B) is the number of occurrences of phrase AB divided bythe total number of two-word phrases in the corpus. The remainingprobabilities are defined as follows:p(˜A)=1−p(A);  (Eq. 2)p(˜B)=1−p(B);  (Eq. 3)p(A,˜B)=p(A)−p(A,B);  (Eq. 4)p(˜A,B)=p(B)−p(A,B); and  (Eq. 5)p(˜A,˜B)=1−p(A,B)−p(˜A,B)−p(A,˜B).  (Eq. 6)

It should be noted that in general, p(A), p(B), and p(A, B) are expectedto be quite small, e.g., on the order of 10⁻⁵ or even smaller.Consequently, the most significant term in Eq. (1) is the x_(i)=A,y_(j)=B term. Thus, in some embodiments, the mutual information metriccan be simplified to the form:

$\begin{matrix}{I = \;{{p\left( {A,B} \right)}\log{\frac{p\left( {A,B} \right)}{{p(A)}{p(B)}}.}}} & \left( {{Eq}.\mspace{14mu} 7} \right)\end{matrix}$

Similar formulas can be applied for mutual information among three wordsor more generally among n words for any n>1.

Given a phrase AB that occurs at least once, the mutual informationcomputed using Eq. (1) will always be nonzero; it will be small if thewords are unrelated (i.e., the phrase AB occurs only by chance) andlarge if the two are closely related (i.e., the phrase AB occurssignificantly more often than chance would suggest).

Using this approach, a mutual information score can be calculated foreach phrase. Other statistical measures such as a conventionalchi-square test can also be used to measure the correlation betweenwords in a phrase to identify phrases where the words occur togethermore frequently than expected by chance.

At block 318, the mutual information score (or other statisticalmeasure) can be used to prune the phrase list, thereby identifyingmeaningful phrases. For example, the phrases can be sorted in descendingorder of mutual information score and a threshold applied to select themeaningful phrases to be used in clustering. The threshold can be set asa limit on the number of phrases retained (e.g., the 50 phrases withhighest scores) or as a minimum mutual information score, which can betuned to provide approximately a desired number of meaningful phrases.

At block 320, an importance score, or weight, can be computed for eachmeaningful phrase. In general, words and phrases that occur inrelatively few documents are considered more useful for clustering thancommon words and phrases. Accordingly, in one embodiment, the weightassigned to a meaningful phrase can be log (1/D), where D is the numberof documents in which the phrase occurs at least once. Other weights canalso be used.

At block 322, process 300 ends. The meaningful phrases can be stored inphrase list 134 (FIG. 1) and used in clustering, e.g., as describedabove.

Phrase List Management

As noted above, one of the difficulties in automatic phrase extractionis managing the number of candidate phrases. Most phrases extracted fromdocuments are uninteresting conjunctions of words, such as “quickbrown,” but since the distinction between meaningful phrases anduninteresting conjunctions of words is not made in advance, suchconjunctions should be tracked. Any number of phrases can be tracked,given enough storage; however, in today's computer systems, providingsufficient storage generally requires using magnetic disk, which isslower than semiconductor memory devices. Thus, in some embodiments,optimization techniques can be used to reduce the number of phrases forwhich information is stored.

FIG. 5 illustrates an optimized phrase list structure 500 according toan embodiment of the invention. Phrase list structure 500 includes twolists: “list A” 502, which stores phrases that have occurred exactlyonce in the portion of the corpus analyzed so far, and “list B” 504,which stores phrases that have occurred at least twice and keeps a count(count) of the number of occurrences of each phrase. In someembodiments, list B also keeps a count (doc_count) of the number ofdocuments in which the phrase occurred at least once. List A 502 has apredetermined maximum size, e.g., one million entries. List B does nothave a maximum size. The first time a phrase occurs, it is added to listA; if the phrase occurs again, it can be moved (promoted) to list B. Iflist A fills up and a new phrase occurs for the first time, the newphrase replaces a previous phrase from list A. The phrase to be replacedcan be selected randomly, pseudorandomly, or according to some otherrule, e.g., replacing the least recently encountered phrase on the list.

FIG. 6 is a flow diagram of a process 600 for managing phrase liststructure 500 of FIG. 5 according to an embodiment of the invention.Process 600 can be used, e.g., at block 312 of process 300 (FIG. 3) toupdate the phrase list.

Process 600 starts (block 602) with a document to be processed. At block604, a phrase from the document is identified. At block 606, the phraseis checked against list B 504 to determine whether the phrase is presentin list B 504. If so, then the count is incremented at block 608.Incrementing the count can include incrementing the occurrence count andalso, if this is the first occurrence of the phrase in the currentdocument, incrementing the document count. Process 600 can then proceedto the next phrase at block 610.

If the phrase is not in list B 504, then at block 612, process 600determines whether the phrase appears in list A 502. If so, then atblock 614, the phrase is promoted by being removed from list A 502 andadded to list B 504. The counter and document counter can be initializedappropriately, and process 600 can proceed to the next phrase at block610.

If the phrase is not in list B 504 or list A 502, then at block 616,process 600 determines if list A 502 is full, i.e., whether it containsthe predetermined maximum number of phrases. If not, then the phrase isadded to list A 502 at block 618. If list A 500 is full, then at block620, a phrase is selected to be removed from list A (e.g., using randomselection or another technique), and at block 622 the new phrase isadded to list A. Thereafter, process 600 can proceed to the next phraseat block 610. When all phrases in the current document have beenprocessed, process 600 ends (block 624).

Process 600 can be used for each document in the corpus, with lists Aand B growing and being continually updated as additional documents areprocessed.

Process 600 provides for a tradeoff between tracking every possiblephrase and performance of the phrase-identification algorithm. It isassumed that n-tuples that occur only once in a large corpus areunlikely to be meaningful phrases; thus, such n-tuples can be droppedfrom consideration. However, it is not possible to know in advance ofn-tuple extraction which n-tuples will occur more than once. Since thelist of single-occurrence candidate phrases is limited in size, it ispossible that an n-tuple would be dropped from that list only to occuragain later. However, if the probability of a particular phrase isgreater than one divided by the maximum number of entries on the list ofsingle-occurrence phrases (one in one million according to oneembodiment), it is very likely that this phrase would be seen againbefore it is dropped. For a sufficiently large list of single-occurrencephrases, it is unlikely that a phrase that occurs often enough to bemeaningful would be dropped from the list. By adjusting the maximum sizeof the single-occurrence list, it is possible to ensure (with highprobability) that all phrases above a certain probability of occurrencewill be promoted to the list of multiple-occurrence phrases, and it willbe possible to make accurate estimates of the probability that thesephrases are not chance conjunctions. Some single-occurrence phrases willbe lost, but such phrases are not likely to be useful for clustering,and the savings in storage requirements is considerable. The risk oflosing a useful phrase can be balanced against storage requirements byadjusting the size limit of the single-occurrence list.

This approach can be generalized to more than two lists. For example,there could be a list of phrases that occur once, a list of phrases thatoccur twice, and a list of phrases that occur three or more times. Whenthe first list fills up, a phrase is removed; when the second list fillsup, a phrase is demoted to the first list.

In some embodiments, all phrases of two or more words are managed usingone set of lists. In another embodiment, separate sets of lists can beprovided for phrases of different lengths; thus, there can be a pair oflists for two-word phrases, a separate pair of lists for three-wordphrases, and so on up to the maximum phrase length considered.

Use of Phrases in Document Vectors and Clustering

Referring back to FIG. 2, after a set of meaningful phrases has beenselected, document vectors are constructed using the words andmeaningful phrases. A document vector can be constructed by arranging alist of words and phrases (in an arbitrary order) to define vectorcomponents and populating each component with a count of the number ofoccurrences of the corresponding word or phrase. The counts can beweighted, e.g., based on importance of the term. In some embodiments,the weight is given by log (1/D), where D is the number of documents inwhich the term occurs at least once. Other weights can also be used. Thecomponent terms and weights are the same for all documents; only thecounts differ.

In some embodiments, the number of phrases included in the documentvector can be limited, e.g., to a maximum of 50 phrases; the number ofwords can be but need not be limited.

Clustering can proceed according to conventional clustering techniquesbased on comparing document vectors. Clustering can produce, e.g., ahierarchical set of clusters, and clusters can be defined based onfeatures of the document vector as is known in the art. The fact thatsome components of the document vectors correspond to phrases ratherthan single words need not affect clustering operations.

Naming Clusters Using Words and Phrases

In block 210 of FIG. 2, a name is assigned to each cluster. In someembodiments, a cluster name is automatically assigned, and the name cansimply consist of a list of the M (e.g., 3 or 5) most frequentlyoccurring terms (words or phrases) in the cluster. It is possible thatsome or all of the most frequently occurring words are also within themost frequently occurring phrases. Thus, the top three terms for aparticular cluster might be (“computer,” “security,” “computersecurity”). For another cluster, the top three might be (“New York,”“Yankees,” “New York Yankees”).

Where a word appears in two different terms, having both terms in thecluster name may not be desirable. Accordingly, in some embodiments,once the top terms are selected, a further comparison is made todetermine whether any words in the list are also in phrases in the listor whether any phrases in the list are in longer phrases that are alsoin the list. If this is the case, one of the two terms can be eliminatedand optionally replaced with a different term.

FIG. 7 is a flow diagram of a process 700 for automatically generating acluster name according to an embodiment of the present invention.Process 700 starts (block 702) when a cluster has been formed, e.g.,using clustering algorithms that incorporate words and meaningfulphrases. At block 704, the most frequently-occurring terms areidentified, and at block 706, these terms are sorted into a listaccording to the weights assigned thereto. (As noted above, the weightcan be log (1/D) or some other weight.) At block 708, it is determinedwhether the list contains any repeated terms. For example, a word mayappear in the list on its own and as part of a phrase, or a phrase mayappear on its own and as part of a longer phrase. If there are norepeated terms, then the top weighted terms on the list (e.g., 3 or 5terms) can be selected as the cluster name at block 710 before process700 ends at block 712.

If, however, there are repeated terms, the list is pruned so that onlyone instance of the term appears. For example, at block 714 the weight(w_(s)) of the shorter term (i.e., the term with fewer words) iscompared to the weight w_(l) of the longer term. At block 716, athreshold test is applied to the weights. The threshold test can favorthe longer term (e.g., “New York Yankees” over “Yankees”) unless theshorter term outweighs it by a significant margin. In one suchembodiment, the threshold is set such that the longer term is selectedunless w_(s) is greater than twice w_(l); other tests can besubstituted.

If the threshold test is met, the longer term is removed from the list(block 718); otherwise, the shorter term is removed (block 720). Ifrepeated terms still remain (block 722), process 700 returns to block708 for further pruning of the list. Once all repeated terms have beenremoved (i.e., no word occurs in more than one term), the top weightedterms can be selected at block 710.

It will be appreciated that process 700 is illustrative and thatvariations and modifications are possible. Steps described as sequentialmay be executed in parallel, order of steps may be varied, and steps maybe modified, combined, added or omitted. For instance, when pruning,single words that appear in phrases can be pruned, followed by two-wordphrases that appear in longer phrases, and so on. Criteria for selectingbetween shorter and longer terms can also be varied.

Further Embodiments

While the invention has been described with respect to specificembodiments, one skilled in the art will recognize that numerousmodifications are possible. For instance, the particular statisticalmetrics for distinguishing meaningful phrases from chance occurrencescan be varied; any metric that is based on the frequency of occurrenceof a particular sequence of words and that measures the likelihood ofthat frequency occurring by chance can be used.

The phrase list management technique described herein is alsoillustrative and can be modified. The limits on the list(s) with limitedsize can be selected as desired, based on tradeoffs between availablestorage and performance. In general, longer lists reduce the probabilityof a semantically meaningful phrase going unrecognized; however, longerlists require more storage space and can increase the time required todetermine whether a given word sequence is already on the list.

As described above, embodiments of the present invention may beimplemented as computer programs. Such programs may be encoded onvarious computer readable media for storage and/or transmission;suitable media include magnetic disk or tape, optical storage media suchas compact disc (CD) or DVD (digital versatile disc), flash memory, andthe like. Computer readable media encoded with the program code may bepackaged with a device (e.g., microprocessor) capable of executing theprogram or provided separately from such devices.

In addition, while the embodiments described above may make reference tospecific hardware and software components, those skilled in the art willappreciate that different combinations of hardware and/or softwarecomponents may also be used and that particular operations described asbeing implemented in hardware might also be implemented in software orvice versa.

Circuits, logic modules, processors, and/or other components may beconfigured to perform various operations described herein. Those skilledin the art will appreciate that, depending on implementation, suchconfiguration can be accomplished through design, setup,interconnection, and/or programming of the particular components andthat, again depending on implementation, a configured component might ormight not be reconfigurable for a different operation. For example, aprogrammable processor can be configured by providing suitableexecutable code; a dedicated logic circuit can be configured by suitablyconnecting logic gates and other circuit elements; and so on.

Thus, although the invention has been described with respect to specificembodiments, it will be appreciated that the invention is intended tocover all modifications and equivalents within the scope of thefollowing claims.

What is claimed is:
 1. A method of extracting phrases from a corpus ofdocuments, the method comprising: generating, by at least one processor,a set of candidate phrases from the documents in the corpus, whereineach candidate phrase corresponds to a group of two or more words thatoccur consecutively in at least one of the documents in the corpus, eachcandidate phrase having an occurrence count associated therewith;computing, by the at least one processor, a statistical metric for eachcandidate phrase based at least in part on the occurrence count, whereinthe statistical metric indicates a likelihood of the words within thecandidate phrase occurring consecutively by chance; selecting, by the atleast one processor, a plurality of the candidate phrases as meaningfulphrases, the selection being based on the statistical metric; computing,by the at least one processor, a phrase weight for each meaningfulphrase, the phrase weight being based on a number of documents in thecorpus that contain the meaningful phrase; and forming, by the at leastone processor, clusters of the documents using the phrase weights. 2.The method of claim 1, wherein the statistical metric is a mutualinformation metric.
 3. The method of claim 1, wherein generating the setof candidate phrases includes: extracting a group of consecutive wordsfrom one of documents in the corpus; determining whether the group ofconsecutive words appears as one of the candidate phrases in a firstlist of candidate phrases; and in response to determining that the groupof consecutive words appears as one of the candidate phrases in thefirst list, incrementing an occurrence count associated with the one ofthe candidate phrases.
 4. The method of claim 3, further comprising: inresponse to determining that the group of consecutive words does notappear as one of the candidate phrases in the first list; determiningwhether the group of consecutive words appears as one of the candidatephrases in a second list of candidate phrases; and in response todetermining that the group of consecutive words appears as one of thecandidate phrases in the second list, promoting the one of the candidatephrases to the first list.
 5. The method of claim 4, further comprising:in response to determining that the group of consecutive words does notappear as one of the candidate phrases in the first list and the secondlist, adding the group of consecutive words as a new candidate phrase tothe second list.
 6. The method of claim 1, further comprising:constructing, by the processor, a document vector for each of aplurality of documents from the corpus, the document vector including afirst plurality of components corresponding to words and a secondplurality of components corresponding to at least some of the meaningfulphrases, the second plurality of components being based on the phraseweights for the at least some meaningful phrases, and wherein formingthe clusters comprises comparing the document vectors.
 7. The method ofclaim 6, further comprising assigning a name to each of the clusters,wherein assigning a name to each of the clusters includes: identifying,as candidate terms, a plurality of most frequently occurring terms forthe cluster, wherein each term is either a word or a meaningful phrase;determining whether any word appears in more than one of the candidateterms; in response to determining that a word appears in more than oneof the candidate terms, selecting only one of the terms in which theword appears as a candidate term; and thereafter selecting some of thecandidate terms to be included in the name for the cluster, theselection being based on weights associated with the candidate terms,wherein the name includes a plurality of terms and wherein no wordappears in more than one of the plurality of terms in the name.
 8. Themethod of claim 1, wherein the phrase weight for a meaningful phrase isbased on 1/D, wherein D is the number of documents in the corpus thatcontain the meaningful phrase.
 9. A non-transitory computer readablestorage medium containing program instructions, which when executed by acomputer system cause the computer system to: generate a set ofcandidate phrases from documents in a corpus, wherein each candidatephrase corresponds to a group of two or more words that occurconsecutively in at least one of the documents in the corpus; compute amutual information metric for each candidate phrase based on one or moreoccurrences of the candidate phrase and one or more separate occurrencesof the words within the candidate phrase; select a plurality ofmeaningful phrases from the set of candidate phrases, the selectionbeing based on the mutual information metric; compute a phrase weightfor each of the meaningful phrases, the phrase weight being based on anumber of documents in the corpus that contain a correspondingmeaningful phrase; and form clusters of the documents using the phraseweights.
 10. The computer readable storage medium of claim 9, whereinthe instructions when executed cause the computer system to furtherassign a name to each of the clusters, wherein assigning the name to atleast one of the dusters includes: identifying, as candidate terms, aplurality of most frequently occurring terms for the cluster, whereineach term is either a word or a meaningful phrase; determining whetherany word appears in more than one of the candidate terms; in response todetermining that a word appears in more than one of the candidate terms,selecting only one of the terms in which the word appears as a candidateterm; and thereafter selecting some of the candidate terms to beincluded in the name for the cluster, the selection being based onweights associated with the candidate terms, wherein the name includes aplurality of terms and wherein no word appears in more than one of theplurality of terms in the name.
 11. The computer readable storage mediumof claim 9, wherein computing the mutual information metric includes, inresponse to determining that the candidate phrase corresponds to a groupof two consecutive words, computing a mutual information metric (I)using a formula:${I = \;{{p\left( {A,B} \right)}\log\frac{p\left( {A,B} \right)}{{p(A)}{p(B)}}}},$wherein: p(A) represents a probability that a first word of a lair ofconsecutive words in a document is a first word of the candidate phrase;p(B) represents a probability that a second word of a pair ofconsecutive words a document is a second word of the candidate phrase;and p(A, B) represents a probability that a pair of consecutive words ina document corresponds to the candidate phrase.
 12. Thecomputer-readable storage medium of claim 9, wherein the instructionswhen executed cause the computer system to further: construct a documentvector for each of the documents, the document vector including a firstplurality of components corresponding to words and a second plurality ofcomponents corresponding to at least some of the meaningful phrases, thesecond plurality of components being based on the phrase weights for theat least some meaningful phrases, and wherein forming the clusterscomprises comparing the document vectors.
 13. A computer systemcomprising; a storage subsystem; and at least one processor coupled tothe storage subsystem, the at least one processor being configured to:extract and store in the storage subsystem a set of candidate phrasesfrom a corpus of documents, wherein each candidate phrase corresponds toa group of two or more words that occur consecutively in at least one ofthe documents in the corpus; compute a statistical metric for each ofthe candidate phrases based on occurrence count data indicating a numberof occurrences of the candidate phrase in documents of the corpus and anumber of occurrences of the words making up the candidate phrase,wherein the statistical metric indicates a likelihood of the wordswithin the candidate phrase occurring consecutively chance; selectphrases from the set of candidate phrases as meaningful phrases, theselection being based on the statistical metric; compute a phrase weightfor each of the meaningful phrases, the phrase weight being based on anumber of documents in the corpus that contain a correspondingmeaningful phrase; and form clusters of the documents using the phraseweights.
 14. The computer system of claim 13, wherein the storagesubsystem is configured to store a first list of candidate phrases and asecond list of candidate phrases, wherein each candidate phrase in thefirst list of candidate phrases has a count value associated therewith,and wherein the at least one processor is further configured such thatextracting the set of candidate phrases comprises: extracting a group oftwo or more consecutive words from one of the documents in the corpus;determining whether the group of two or more consecutive words appearsas one of the candidate phrases in the first list of candidate phrases;and in response to determining that the group of two or more consecutivewords appears as one of the candidate phrases in the first list,incrementing the count value associated with the one of the candidatephrases.
 15. The computer system of claim 14, wherein extracting the setof candidate phrases further comprises: in response to determining thatthe group of two or more consecutive words does not appear as one of thecandidate phrases in the first list: determining whether the group oftwo or more consecutive words appears as one of the candidate phrases inthe second list; in response to determining that the group of two ormore consecutive words appears as one of the candidate phrases in thesecond list, moving the one of the candidate phrases from the secondlist to the first list; and in response to determining that the group oftwo or more consecutive words does not appear as one of the candidatephrases in the second list, adding the group of consecutive words as anew candidate phrase to the second list.
 16. The computer system ofclaim 13, wherein the at least one processor is further configured tocompute the statistical metric as a mutual information metric (I) usinga formula:${I = \;{{p\left( {A,B} \right)}\log\frac{p\left( {A,B} \right)}{{p(A)}{p(B)}}}},$wherein: p(A) represents a probability that a first Word of a pair ofconsecutive words in a document is a first word of the candidate phrase;p(B) represents a probability that a second word of a pair ofconsecutive words in a document is a second word of the candidatephrase; and p(A, B) represents a probability that a pair of consecutivewords in a document corresponds to the candidate phrase.
 17. Thecomputing system of claim 13, wherein the at least one processor isconfigured to further construct a document vector for each of thedocuments, the document vector including a first plurality of componentscorresponding to words and a second plurality of componentscorresponding to at least some of the meaningful phrases, the secondplurality of components being based on the phrase weights for the atleast some meaningful phrases, and wherein forming the clusterscomprises comparing the document vectors.
 18. The computing system ofclaim 13, wherein the at least one processor is to further assign acluster name to at least one of the document clusters by: identifying,as candidate terms, a plurality of most frequently occurring terms forthe cluster, wherein each term is either a word or a meaningful phrase;determining whether any word appears in more than one of the candidateterms; in response to determining that a word appears in more than oneof the candidate terms, selecting only one of the terms in which theword appears as a candidate term; and thereafter selecting some of thecandidate terms to be included in the name for the cluster, theselection being based on weights associated with the candidate terms,wherein the name includes a plurality of terms and wherein no wordappears in more than one of the plurality of terms in the name.